Node: latitude/longitude storage #4292

milaaraujo · 2018-12-13T05:02:53Z

milaaraujo · 2018-12-13T05:14:17Z

@jywarren, wherein the code do we save the latitude/longitude as tags?

Okay, let's suppose we have successfully created new columns in the node table and now we have latitude/longitude as floats and their precision. But all old nodes just have the latitude/longitude saved as tags... So we can not change all the current methods, otherwise, we break what already exists.

What is your suggestion regarding this? Should we try to recursively retrieve lat/lon of the old nodes and save in these new columns? Create new endpoints?

jywarren · 2018-12-14T00:32:28Z

We can make a migration which would select all tags starting with lat and Lon, and add the extra columns. How's that sound?

…

On Thu, Dec 13, 2018, 12:14 AM Camila Araújo ***@***.*** wrote: @jywarren <https://github.com/jywarren>, wherein the code do we save the latitude/longitude as tags? Okay, let's suppose we have successfully created new columns in the node table and now we have latitude/longitude as floats and their precision. But all old nodes just have the latitude/longitude saved as tags... So we can not change all the current methods, otherwise, we break what already exists. What is your suggestion regarding this? Should we try to recursively retrieve lat/lon of the old nodes and save in these new columns? Create new endpoints? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4292 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ7bIGVzekP3aU7lUF7XFQ6gBHht8ks5u4eIqgaJpZM4ZQ7pm> .

milaaraujo · 2018-12-14T05:37:52Z

Sounds good... I just don’t know how to do it! Do you have an example?

jywarren · 2018-12-16T06:26:43Z

Yes, sorry - take a look at some of the migrations in /db/migrate/... -- esp. ones that iterate through all of one type of model. I'm sure we've done some of this for all Users, for example. This is run upon deploying the code, as a kind of installation/update script. Make sense?

milaaraujo · 2018-12-18T09:08:06Z

Hey @jywarren, working with migrations is harder than I thought it would be... I was able to transfer the value of the tags to the new columns, but the problem now is that I do not know how to maintain the float precision!

MariaDB [publiclab_dev]> select nid, `latitude`, `longitude`, `precision` from node WHERE latitude != 0;
+-----+----------+-----------+-----------+
| nid | latitude | longitude | precision |
+-----+----------+-----------+-----------+
|  10 |  8.81467 |   17.3655 |        15 |
|  11 |   37.119 |   77.0585 |        14 |
|  12 |  16.0045 |   69.3345 |        12 |
|  13 |  15.8983 |   44.8083 |        14 |
|  14 |  32.9407 |   14.6519 |        15 |
|  15 |  29.9538 |   27.6056 |        14 |
|  16 |  61.4815 |   31.1145 |        15 |
|  17 |  74.9755 |   15.2274 |        15 |
|  18 |  35.6731 |   45.0279 |        15 |
|  19 |  66.5724 |   75.7069 |        13 |
|  20 |  64.0738 |   74.0497 |        14 |
|  21 |  4.28079 |   48.1806 |        13 |
|  22 |  79.3643 |   8.03875 |        15 |
|  23 |  10.6207 |   26.9991 |        15 |
|  24 |  51.1041 |   21.7141 |        15 |
|  25 |  69.2186 |   1.40931 |        16 |
|  26 |  35.0613 |  0.326677 |        16 |
|  27 |  3.52222 |   50.2646 |        14 |
|  28 |  51.3302 |   34.8903 |        14 |
|  29 |  28.0015 |   15.9763 |        15 |
|  30 |  68.8966 |   17.9573 |        15 |
|  31 |  70.5209 |   18.5898 |        14 |
|  32 |  22.0404 |   7.71761 |        15 |
|  33 |  30.8439 |     48.84 |        15 |
|  34 |  74.9298 |    23.681 |        15 |
|  35 |  53.3268 |   48.9159 |        15 |
|  36 |  43.8986 |   36.4432 |        15 |
|  37 |  17.1721 |   33.6177 |        14 |
|  38 |  50.7842 |   56.5169 |        15 |
|  39 |  55.7599 |   17.1723 |        14 |
|  40 |  55.1529 |   45.7677 |        14 |
|  41 |  68.8346 |   63.8693 |        14 |
|  42 |  1.08384 |   37.9133 |        14 |
|  43 |  23.0147 |   48.0152 |        15 |
+-----+----------+-----------+-----------+

I've already tried using decimal and a bunch of other things, but nothing worked. Do you have any suggestion?

jywarren · 2018-12-18T13:22:04Z

Ah, this is actually why we are storing precision separately - float precision is not precisely controllable. But we definitely should make the lat/lon fields larger - there's a way to increase the precision of those fields ... let me see; something like this: db/migrate/20130114213934_add_profile_tags.rb: #add_column :users, :lat, :decimal, :precision => 20, :scale => 10, :default => 0.0 db/migrate/20130114213934_add_profile_tags.rb: #add_column :users, :lon, :decimal, :precision => 20, :scale => 10, :default => 0.0 Make sense? That way we store an excess of precision (the decimals will be inconsistent at greater than 20 precision) but we use the integer precision field to truncate them during use.

…

On Tue, Dec 18, 2018 at 4:08 AM Camila Araújo ***@***.***> wrote: Working with migrations is harder than I thought it would be... I was able to transfer the value of the tags to the new columns, but the problem now is that I do not know how to maintain the float precision! MariaDB [publiclab_dev]> select nid, `latitude`, `longitude`, `precision` from node WHERE latitude != 0; +-----+----------+-----------+-----------+ | nid | latitude | longitude | precision | +-----+----------+-----------+-----------+ | 10 | 8.81467 | 17.3655 | 15 | | 11 | 37.119 | 77.0585 | 14 | | 12 | 16.0045 | 69.3345 | 12 | | 13 | 15.8983 | 44.8083 | 14 | | 14 | 32.9407 | 14.6519 | 15 | | 15 | 29.9538 | 27.6056 | 14 | | 16 | 61.4815 | 31.1145 | 15 | | 17 | 74.9755 | 15.2274 | 15 | | 18 | 35.6731 | 45.0279 | 15 | | 19 | 66.5724 | 75.7069 | 13 | | 20 | 64.0738 | 74.0497 | 14 | | 21 | 4.28079 | 48.1806 | 13 | | 22 | 79.3643 | 8.03875 | 15 | | 23 | 10.6207 | 26.9991 | 15 | | 24 | 51.1041 | 21.7141 | 15 | | 25 | 69.2186 | 1.40931 | 16 | | 26 | 35.0613 | 0.326677 | 16 | | 27 | 3.52222 | 50.2646 | 14 | | 28 | 51.3302 | 34.8903 | 14 | | 29 | 28.0015 | 15.9763 | 15 | | 30 | 68.8966 | 17.9573 | 15 | | 31 | 70.5209 | 18.5898 | 14 | | 32 | 22.0404 | 7.71761 | 15 | | 33 | 30.8439 | 48.84 | 15 | | 34 | 74.9298 | 23.681 | 15 | | 35 | 53.3268 | 48.9159 | 15 | | 36 | 43.8986 | 36.4432 | 15 | | 37 | 17.1721 | 33.6177 | 14 | | 38 | 50.7842 | 56.5169 | 15 | | 39 | 55.7599 | 17.1723 | 14 | | 40 | 55.1529 | 45.7677 | 14 | | 41 | 68.8346 | 63.8693 | 14 | | 42 | 1.08384 | 37.9133 | 14 | | 43 | 23.0147 | 48.0152 | 15 | +-----+----------+-----------+-----------+ I've already tried using decimal and a bunch of other things, but nothing worked. Do you have any suggestion? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4292 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ_j3PmWdL3GVY9H5MR_zpn_LnluLks5u6LB2gaJpZM4ZQ7pm> .

milaaraujo · 2018-12-19T04:48:37Z

@milaaraujo jywarren, I think it is working now:

MariaDB [publiclab_dev]> select nid, `latitude`, `longitude`, `precision` from node WHERE latitude != 0;
+-----+----------------------+----------------------+-----------+
| nid | latitude             | longitude            | precision |
+-----+----------------------+----------------------+-----------+
|  10 |  8.81466577639189500 | 17.36552043355546400 |        15 |
|  11 | 37.11897112103856000 | 77.05847778914394000 |        14 |
|  12 | 16.00447867791259400 | 69.33453058937500000 |        12 |
|  13 | 15.89831741202378000 | 44.80827750655291000 |        14 |
|  14 | 32.94068717446902000 | 14.65193661308239500 |        15 |
|  15 | 29.95376561862603600 | 27.60562661230719000 |        14 |
|  16 | 61.48149305525445000 | 31.11446611242257600 |        15 |
|  17 | 74.97548098431554000 | 15.22738638597561900 |        15 |
|  18 | 35.67314305234031000 | 45.02784955920646600 |        15 |
|  19 | 66.57236621979462000 | 75.70690324643560000 |        13 |
|  20 | 64.07378864856013000 | 74.04971060756745000 |        14 |
|  21 |  4.28078941902575700 | 48.18063408873650000 |        13 |
|  22 | 79.36429574980298000 |  8.03875285724650700 |        15 |
|  23 | 10.62070366305438700 | 26.99905390488142500 |        15 |
|  24 | 51.10410583917507000 | 21.71413744558739400 |        15 |
|  25 | 69.21861314079344000 |  1.40930898888311340 |        16 |
|  26 | 35.06131641427838000 |  0.32667663394520650 |        16 |
|  27 |  3.52222175763125200 | 50.26456276669801000 |        14 |
|  28 | 51.33020686356304000 | 34.89026572929842000 |        14 |
|  29 | 28.00153071886246200 | 15.97633299989550600 |        15 |
|  30 | 68.89663788161903000 | 17.95726150145208400 |        15 |
|  31 | 70.52084892415182000 | 18.58976682700557000 |        14 |
|  32 | 22.04035236852823000 |  7.71761056821617300 |        15 |
|  33 | 30.84388551371620200 | 48.84004264550349500 |        15 |
|  34 | 74.92979837188261000 | 23.68095976143603600 |        15 |
|  35 | 53.32680235652098000 | 48.91592381883043600 |        15 |
|  36 | 43.89864834428540000 | 36.44322708705179500 |        15 |
|  37 | 17.17214480094377600 | 33.61767747472112000 |        14 |
|  38 | 50.78423478199642500 | 56.51689789528768400 |        15 |
|  39 | 55.75985866867546000 | 17.17228052625802000 |        14 |
|  40 | 55.15285948515678000 | 45.76772749324683000 |        14 |
|  41 | 68.83460642210348000 | 63.86929302653418000 |        14 |
|  42 |  1.08383579651352600 | 37.91330968020314000 |        14 |
|  43 | 23.01465406797677800 | 48.01518821885685600 |        15 |
+-----+----------------------+----------------------+-----------+

What do you think? How do we merge migrations?

jywarren

Very nice. Ready to merge?

jywarren · 2018-12-19T15:40:59Z

Ah, I think one last step is to update our schema.rb.example to the latest schema - with the extra fields you can copy in from your newly generated schema. The timestamp checksum at the top of the file will also have to be updated, and once these are added, the tests should pass.

jywarren · 2018-12-19T15:41:33Z

For an example, try looking at the history for the schema.rb.example file!

plotsbot · 2018-12-19T17:12:44Z

	1 Warning
⚠️	New migrations added. Please update `schema.rb.example` by overwriting it with a copy of the up-to-date `db/schema.rb`. Also, be aware to preserve the MySQL-specific conditions for full-text indices.

	1 Message
📖	@milaaraujo Thank you for your pull request! I’m here to help with some tips and recommendations. Please take a look at the list provided and help us review and accept your contribution! And don’t be discouraged if you see errors – we’re here to help.

Generated by 🚫 Danger

milaaraujo · 2018-12-19T17:25:22Z

Done!

milaaraujo · 2018-12-19T18:19:31Z

Hummmm, all checks have passed, but we have a warning: New migrations added. Please update schema.rb.example by overwriting it with a copy of the up-to-date db/schema.rb. Also, be aware to preserve the MySQL-specific conditions for full-text indices.

I copied just the extra fields and the version from my newly generated schema to the schema.rb.example - not the whole file. Is it okay?

milaaraujo · 2018-12-20T04:23:39Z

Ok, now I copied all the db/schema.rb to schema.rb.example. Can we merge it?

jywarren · 2018-12-22T02:15:15Z

db/schema.rb.example

+    t.string "slug"
+    t.integer "legacy_views", default: 0
+    t.integer "views", default: 0
+    t.decimal "latitude", precision: 20, scale: 17


OK, sorry i have been traveling, but this and the line below I think are the only lines needed except for the timestamp at top. The rest is mostly whitespace changes as well as some encoding changes we don't actually want to modify in this PR. Thanks, Camila!

No problem :)

Is it ok now?

jywarren · 2018-12-23T17:54:33Z

Awesome! Great work. 👍 👍 👍

jywarren · 2019-01-03T20:49:47Z

db/migrate/20181219043340_add_fields_to_node.rb

+        update "UPDATE `node` SET `longitude` = '" + long.to_s + "' WHERE nid = '" + node_id.to_s + "'"
+      else
+        lati = tag["name"].gsub("lat:","")
+        update "UPDATE `node` SET `latitude` = '" + lati + "' WHERE nid = '" + node_id.to_s + "'"


Hi @milaaraujo it looks like this didn't run correctly on production for some reason -- odd that it didn't fail on stable, but i'm looking into it now. I think in general we may want to stick to non-raw SQL in the future if possible as there may be more safeguards... but i don't really know that this is the issue right now:

Mysql2::Error: Incorrect decimal value: 'false' for column 'latitude' at row 1 ActiveRecord::StatementInvalid: Mysql2::Error: Incorrect decimal value: 'false' for column 'latitude' at row 1: UPDATE `node` SET `latitude` = 'false' WHERE nid = '2120'

I wonder if some of the lat/lon tags are not supplying numbers!!??

Aha! I think this tag exists, though not sure where: https://publiclab.org/tag/lat:false

OK, i'll try to manually remove it and re-run the migration. If that doesn't work, we may want to amend the migration to only run if the latitude/longitude values are actually numbers...

OK, deleted lat:false as a tag. I tried scanning on the console for any other non-numerical ones and wasn't able to find any. Hopefully it'll run properly this time, restarting! Sorry for the scare! This is publishing to production now, exciting!!!

Did it work?

jywarren · 2019-01-03T23:51:51Z

Yes! Whew!!! All good and published now!

…

On Thu, Jan 3, 2019, 6:49 PM Camila Araújo ***@***.*** wrote: ***@***.**** commented on this pull request. ------------------------------ In db/migrate/20181219043340_add_fields_to_node.rb <#4292 (comment)>: > + def up + add_column :node, :latitude, :decimal, :precision => 20, :scale => 17 + add_column :node, :longitude, :decimal, :precision => 20, :scale => 17 + add_column :node, :precision, :integer + + sql = "SELECT `node`.`nid`,`term_data`.`name` FROM `node` LEFT OUTER JOIN `community_tags` ON `community_tags`.`nid` = `node`.`nid` LEFT OUTER JOIN `term_data` ON `term_data`.`tid` = `community_tags`.`tid` WHERE (term_data.name LIKE 'lat:%' or term_data.name LIKE 'lon:%')" + tags_array = ActiveRecord::Base.connection.exec_query(sql) + tags_array.each do |tag| + node_id = tag["nid"] + if tag["name"].include? 'lon:' + long = tag["name"].gsub("lon:","").to_f + update "UPDATE `node` SET `precision` = '" + decimals(long).to_s + "' WHERE nid = '" + node_id.to_s + "'" + update "UPDATE `node` SET `longitude` = '" + long.to_s + "' WHERE nid = '" + node_id.to_s + "'" + else + lati = tag["name"].gsub("lat:","") + update "UPDATE `node` SET `latitude` = '" + lati + "' WHERE nid = '" + node_id.to_s + "'" Did it work? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4292 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ1XC-Ti4-jJSFwx1Qb9Owp1O9vxuks5u_pb-gaJpZM4ZQ7pm> .

* populating new fields * adding precision/scale * updating example schema

ghost assigned milaaraujo Dec 13, 2018

ghost added the in progress label Dec 13, 2018

milaaraujo mentioned this pull request Dec 13, 2018

Develop latitude/longitude storage for tags #4183

Closed

3 tasks

populating new fields

fbd2293

milaaraujo force-pushed the latitudeLongitudeStorage branch from 4f952ac to fbd2293 Compare December 18, 2018 09:00

adding precision/scale

d54e973

jywarren approved these changes Dec 19, 2018

View reviewed changes

updating example schema

9869cda

milaaraujo changed the title ~~Node: latitude/longitude storage [WIP]~~ Node: latitude/longitude storage Dec 19, 2018

milaaraujo added ready and removed in progress labels Dec 19, 2018

jywarren requested changes Dec 22, 2018

View reviewed changes

milaaraujo force-pushed the latitudeLongitudeStorage branch from b6f95d3 to 9869cda Compare December 23, 2018 08:30

jywarren approved these changes Dec 23, 2018

View reviewed changes

jywarren merged commit d80c885 into publiclab:master Dec 23, 2018

ghost removed the ready label Dec 23, 2018

jywarren mentioned this pull request Dec 23, 2018

Weekly Community Check-In #2 - "Everybody Struggles"! #4369

Closed

milaaraujo deleted the latitudeLongitudeStorage branch December 24, 2018 03:34

jywarren reviewed Jan 3, 2019

View reviewed changes

SrinandanPai pushed a commit to SrinandanPai/plots2 that referenced this pull request May 5, 2019

Node: latitude/longitude storage (publiclab#4292)

97634b2

* populating new fields * adding precision/scale * updating example schema

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Node: latitude/longitude storage #4292

Node: latitude/longitude storage #4292

milaaraujo commented Dec 13, 2018

milaaraujo commented Dec 13, 2018

jywarren commented Dec 14, 2018 via email

milaaraujo commented Dec 14, 2018

jywarren commented Dec 16, 2018

milaaraujo commented Dec 18, 2018 •

edited

Loading

jywarren commented Dec 18, 2018 via email

milaaraujo commented Dec 19, 2018 •

edited

Loading

jywarren left a comment

jywarren commented Dec 19, 2018

jywarren commented Dec 19, 2018

plotsbot commented Dec 19, 2018 •

edited

Loading

milaaraujo commented Dec 19, 2018

milaaraujo commented Dec 19, 2018

milaaraujo commented Dec 20, 2018 •

edited

Loading

jywarren Dec 22, 2018

milaaraujo Dec 23, 2018

jywarren commented Dec 23, 2018

jywarren Jan 3, 2019

jywarren Jan 3, 2019

jywarren Jan 3, 2019

milaaraujo Jan 3, 2019

jywarren commented Jan 3, 2019 via email

Node: latitude/longitude storage #4292

Node: latitude/longitude storage #4292

Conversation

milaaraujo commented Dec 13, 2018

milaaraujo commented Dec 13, 2018

jywarren commented Dec 14, 2018 via email

milaaraujo commented Dec 14, 2018

jywarren commented Dec 16, 2018

milaaraujo commented Dec 18, 2018 • edited Loading

jywarren commented Dec 18, 2018 via email

milaaraujo commented Dec 19, 2018 • edited Loading

jywarren left a comment

Choose a reason for hiding this comment

jywarren commented Dec 19, 2018

jywarren commented Dec 19, 2018

plotsbot commented Dec 19, 2018 • edited Loading

milaaraujo commented Dec 19, 2018

milaaraujo commented Dec 19, 2018

milaaraujo commented Dec 20, 2018 • edited Loading

jywarren Dec 22, 2018

Choose a reason for hiding this comment

milaaraujo Dec 23, 2018

Choose a reason for hiding this comment

jywarren commented Dec 23, 2018

jywarren Jan 3, 2019

Choose a reason for hiding this comment

jywarren Jan 3, 2019

Choose a reason for hiding this comment

jywarren Jan 3, 2019

Choose a reason for hiding this comment

milaaraujo Jan 3, 2019

Choose a reason for hiding this comment

jywarren commented Jan 3, 2019 via email

milaaraujo commented Dec 18, 2018 •

edited

Loading

milaaraujo commented Dec 19, 2018 •

edited

Loading

plotsbot commented Dec 19, 2018 •

edited

Loading

milaaraujo commented Dec 20, 2018 •

edited

Loading